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Concentration of Measure Inequalities for 
Toeplitz Matrices with Applications 

Borhan M. Sanandaji, Tyrone L. Vincent, and Michael B. Wakin 

Abstract 

We derive Concentration of Measure (CoM) inequalities for randomized Toeplitz matrices. These 
inequalities show that the norm of a high-dimensional signal mapped by a Toeplitz matrix to a low- 
dimensional space concentrates around its mean with a tail probability bound that decays exponentially 
in the dimension of the range space divided by a quantity which is a function of the signal. For the 
class of sparse signals, the introduced quantity is bounded by the sparsity level of the signal. However, 
we observe that this bound is highly pessimistic for most sparse signals and we show that if a random 
distribution is imposed on the non-zero entries of the signal, the typical value of the quantity is boimded 
by a term that scales logarithmically in the ambient dimension. As an application of the CoM inequalities, 
we consider Compressive Binary Detection (CBD). 

Index Terms 

Concentration of Measure InequaUties, Compressive ToepUtz Matrices, Compressive Sensing. 

I. Introduction 

A. Overview 

Motivated to reduce the burdens of acquiring, transmitting, storing, and analyzing vast quantities of 
data, signal processing researchers have over the last few decades developed a variety of techniques for 
data compression and dimensionaUty reduction. Unfortunately, many of these techniques require a raw, 
high-dimensional data set to be acquired before its essential low-dimensional structure can be identified, 
extracted, and exploited. In contrast, what would be truly desirable are sensors/operators that require 
fewer raw measurements yet still capture the essential information in a data set. These operators can be 
called compressive in the sense that they act as mappings from a high-dimensional to a low-dimensional 
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space, e.g., X : M — )• M where M < N. Linear compressive operators correspond to matrices having 
fewer rows than columns. Although such matrices can have arbitrary/deterministic entries, randomized 
matrices (those with entries drawn from a random distribution) have attracted the attention of researchers 
due to their universality and ease of analysis. Utilizing such compressive operators to achieve information- 
preserving embeddings of high-dimensional (but compressible) data sets into low-dimensional spaces can 
drastically simpUfy the acquisition process, reduce the needed amount of storage space, and decrease the 
computational demands of data processing. 

Concentration of Measure (CoM) inequalities are one of the leading techniques used in the theoretical 
analysis of randomized compressive linear operators |[T|-|[3|. These inequalities quantify how well a 
random matrix will preserve the norm of a high-dimensional signal when mapping it to a low-dimensional 
space. A typical CoM inequality takes the following form. For any fixed signal a € M^, and a suitable 
random M x N matrix X, the random variable ||Xa||2 will be highly concentrated around its expected 
value, E [HXaHl] , with high probability. Formally, there exist constants ci and C2 such that for any fixed 
a € M^, 

P{\\\Xa\\l - E [\\Xag] \ > eE [\\Xa\\l] } < c,e-'^^'''^^^^\ (1) 

where cq (e) is a positive constant that depends on e G (0, 1). 

CoM inequalities for random operators have been shown to have important implications in signal pro- 
cessing and machine learning. One of the most prominent results in this area is the Johnson-Lindenstrauss 
(JL) lemma, which concerns embedding a finite set of points in a lower dimensional space using a distance 
preserving mapping ||4|. Dasgupta et al. [2| and Achlioptas |3| showed how a CoM inequality of the 
form ([T]) could establish that with high probability, an independent and identically distributed (i.i.d.) 
random compressive operator X G ]^MxN ^ jy) provides a JL-embedding. Specifically, for a given 
e G (0, 1), for any fixed point set Q CR^, 

(1 - e) \\a - b\\l < \\Xa - Xb\\l < (1 + e) \\a - b\\l (2) 

holds with high probability for all a, b £ Q if M = O (e^^log One of the other significant 

consequences of CoM inequalities is in the context of Compressive Sensing (CS) |[5| and the Restricted 
Isometry Property (RIP) ||6|. If a matrix X satisfies ^ for all pairs a, b of K-sparse signals in M^, then 
X is said to satisfy the RIP of order 2K with isometry constant e. Establishing the RIP of order 2K 
for a given compressive matrix X leads to understanding the number of measurements required to have 
exact recovery for any ET-sparse signal a G R^. Baraniuk et al. [7] and Mendelson et al. ijsj showed 
that CoM inequalities can be used to prove the RIP for random compressive matrices. 
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CoM inequalities have been well- studied and derived for unstructured random compressive matrices, 
populated with i.i.d. random entries Q, ||3|. However, in many practical applications, measurement 
matrices possess a certain structure. In particular, when linear dynamical systems are involved, Toeplitz 
and circulant matrices appear due to the convolution process |[9|-|12|. Specifically, consider the linear 
time-invariant (LTI) dynamical system with system finite impulse response a = {ak}^=i- Let x = 
{xk}k=i^^^^ be the applied input sequence. Then the corresponding output is calculated from the time- 
domain convolution of a and x. Supposing the and sequences are zero-padded from both sides, 
each output sample can be written as 

N 

Vk = y^gjXfc-j. (3) 

i=i 

If we keep only M consecutive observations of the system, y = {yfej^^^^i, then (jsjl can be written in 
matrix-vector multiplication format as 

y = Xa, (4) 



where 



X 



XN XAr_i • • • Xl 

Xn+1 Xn • • • X2 



(5) 



XN+M-l Xn^m~2 ■ ■ ■ Xm 

is an M X Toeplitz matrix. If the entries of X are generated randomly, we say X is a random- 
ized Toeplitz matrix. Other types of structured random matrices also arise when dynamical systems 
are involved. For example block-diagonal matrices appear in applications such as distributed sensing 
systems |13| and initial state estimation (observability) of linear systems fTT]. 

In this paper, we consider compressive randomized Toeplitz matrices, derive CoM inequalities, and 



discuss their implications in applications such as sparse impulse response recovery |10|, |15|, |16|. We 
also consider the problem of detecting a deviation in a system's behavior. We show that by characterizing 
the deviation using a particular measure that appears in our CoM inequality, the detector performance 
can be correctly predicted. 



B. Related Work 

Compressive Toeplitz (and circulant) matrices have been previously studied in the context of CS fOl- 



|11|, |15|, |17|, |18|, with applications involving channel estimation, synthetic aperture radar, etc. Tropp 
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et al. 1 17 1 originally considered compressive Toeplitz matrices in an early CS paper that proposed an 
efficient measurement mechanism involving a Finite Impulse Response (FIR) filter with random taps. 
Motivated by applications related to sparse channel estimation, Bajwa et al. |18| studied such matrices 
more formally in the case where the matrix entries are drawn from a symmetric Bernoulli distribution. 
Later they extended this study to random matrices whose entries are bounded or Gaussian-distributed and 
showed that with high probability, M > O (K^log(^)) measurements are sufficient to estabUsh the 



RIP of order 2K for vectors sparse in the time domain p()| , 1 15 1. (It should be noted that the quadratic 
RIP result can also be achieved using other methods such as a coherence argument ||9|, |19|.) Recently, 
using more complicated mathematical tools such as Dudley's inequality for chaos and generic chaining, 
Rauhut et al. |20| showed that with M >0 [K^-^log {N)^-^] measurements the RIP of order 2K will 



hold|i]Note that these bounds compare to M > O (^K log (^)) measurements which are known to suffice 
when X is unstructured p\. 

In this paper, we derive CoM inequalities for Toeplitz matrices and show how these inequalities reveal 
non-uniformity and signal-dependency of the mappings. As one consequence of these CoM inequalities, 
one could use them (along with standard covering number estimates) to prove the RIP for compressive 
Toeplitz matrices. Although the estimate of the required number of measurements would be quadratic in 
terms of sparsity (i.e., M ~ K^) and fall short of the best known estimates described above, studying 
concentration inequalities for Toeplitz matrices is of its own interest and gives insight to other appUcations 
such as the binary detection problem. 

There also exist CoM analyses for other types of structured matrices. For example. Park et al. [13] 
derived concentration bounds for two types of block diagonal compressive matrices, one in which the 
blocks along the diagonal are random and independent, and one in which the blocks are random but 
equalj^ We subsequently extended these CoM results for block diagonal matrices to the observability 



matrices that arise in the analysis of linear dynamical systems 1 14|. 



'After the initial submission of our manuscript, in tfieir very recent worlc, Kralimer et al. |21j showed that the minimal required 
number of measurements scales linearly with K, or formally M > O (^K log (K)^ log (N)^) measurements are sufficient to 
establish the RIP of order 2K. The recent linear RIP result confirms what is suggested by simulations. 

^Shortly after our own development of CoM inequalities for compressive Toeplitz matrices (a preliminary version of Theoremjl] 
appeared in (12|), Yap and Rozell (22| showed that similar inequalities can be derived by extending the CoM results for block 
diagonal matrices. Our Theorem |2j and the associated discussion, however, is unique to this paper. 
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C. Contributions 

In summary, we derive CoM inequalities for randomized Toeplitz matrices. The derived bounds in 
the inequalities are non-uniform and depend on a quantity which is a function of the signal. For the 
class of sparse signals, the introduced quantity is bounded by the sparsity level of the signal while if a 
random distribution is imposed on the non-zero entries of the signal, the typical value of the quantity is 
bounded by a term that scales logarithmically in the ambient dimension. As an application of the CoM 
inequalities, we consider Compressive Binary Detection (CBD). 



II. Main Results 

In this paper, we derive CoM bounds for compressive Toeplitz matrices as given in ([5]) with entries 
{xfcif +*^"^ drawn from an i.i.d. Gaussian random sequence. Our first main result, detailed in Theorem [ij 
states that the upper and lower tail probability bounds depend on the number of measurements M and 
on the eigenvalues of the covariance matrix of the vector a defined as 

^a(O) 7^a(l) ••• TZaiM-l 

na{i) 7^a(o) ••• na{M-2 



P(a) 



(6) 



7^a(M-l) 7^a(M-2) ••• 7^a(0) 

where TZa (r) := J^iLi' ^i^i+r denotes the un-normalized sample autocorrelation function of a G 

Theorem 1: Let a G be fixed. Define two quantities p (a) and /x (a) associated with the eigenvalues 
of the covariance matrix P (a) as p (a) := "||^^|^2 and p (a) := M\\a\\t^ ' where Aj is the i-th eigenvalue 
of P (a). Let y = Xa, where X is a random compressive Toeplitz matrix with i.i.d. Gaussian entries 
having zero mean and unit variance. Noting that E [||y|||] = M||a||2, then for any e € (0, 1), the upper 
tail probability bound is 

P {llylli - M\\a\\l > eM||a||^} < e"^ (7) 
and the lower tail probability bound is 

P {||y||2 - M\\a\\l < -eM||a||2} < e"^^. (8) 



Theorem [ij provides CoM inequalities for any (not necessarily sparse) signal a G M . The significance 
of these results comes from the fact that the tail probability bounds are functions of the signal a, where 
the dependency is captured in the quantities p (a) and p (a). This is not the case when X is unstructured. 
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Indeed, allowing X to have M x N i.i.d. Gaussian entries with zero mean and unit variance (and thus, 
no Toeplitz structure) would result in the concentration inequality (see, e.g., |[3|) 

P{|ll2/ll2 -^^I|a|l2| > e^^llalli} < 2e"^. (9) 

Thus, comparing the bound in Q with the ones in (|7]) and ([8]), one could conclude that achieving the 
same probability bound for Toeplitz matrices requires choosing M larger by a factor of 2p (a) or 2/i (a). 
Typically, when using CoM inequalities such as ^ and ([8]l, we must set M large enough so that both 
bounds are sufficiently small over all signals a belonging to some class of interest. For example, we are 
often interested in signals that have a sparse representation. Because we generally wish to keep M as 
small as possible, it is interesting to try to obtain an upper bound for the important quantities p (a) and 
fi{a) over the class of signals of interest. It is easy to show that for all a € M^, (a) < p{a). Thus, 
we limit our analysis to finding the sharpest upper bound for p (a) when a is i^-sparse. For the sake of 
generality, we allow the signal to be sparse in an arbitrary orthobasis. 

Definition 1: A signal a G is called K -sparse in an orthobasis G G M^^^ if it can be represented 
as a = Gq, where q G is i^-sparse (a vector with K < N non-zero entries). 

We also introduce the notion of K-sparse Fourier coherence of the orthobasis G. This measures how 
strongly the columns of G are correlated with the length L Fourier basis, Fl G C^^^, which has entries 
FL{i,m) = -^=it;(^-i)(™-i), where w = e'^. 

Definition 2: Given an orthobasis G G M^^^ and measurement length M, let L = N + M — 1. The 
K-sparse Fourier coherence of G, denoted uk (G), is defined as 

UK{G):=max\\Fi;^Gs\\2, (10) 

i,S 

where S C {1,2, .. . , A''} is the support set and varies over all possible sets with cardinality l^l = K, 
Gs G R^^^ is a matrix containing the columns of G G M^^^ indexed by the support set S, and 
Fl^ G is a row vector containing the first N entries of the i-th row of the Fourier orthobasis 
Fl G C^^^. Observe that for a given orthobasis G, vk (G) depends on K. 



Using the notion of Fourier coherence, we show in Section IV that for all vectors a G that are 
K-sparse in an orthobasis G G M^^^, 

p{a)<Lul{G), (11) 

where, as above, L = N + M — \. This bound, however, appears to be highly pessimistic for most 
K-sparse signals. As a step towards better understanding the behavior of p{a), we consider a random 
model for a. In particular, we consider a fixed K-sparse support set, and on this set we suppose the K 
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non-zero entries of the coefficient vector q are drawn from a random distribution. Based on this model, 
we derive an upper bound for E [p (a)]. 

Theorem 2: {Upper Bound on E[p(a)]) Let q G be a random i^-sparse vector whose K non- 
zero entries (on an arbitrary support S) are i.i.d. random variables drawn from a Gaussian distribution 
with AA(0, Select the measurement length M, which corresponds to the dimension of P{a), and set 
L = N + M - 1. Let a = Gq where G G R^''^ is an orthobasis. Then 

E [p (a)] < ^I^^Mll (log 2L + 2). (12) 



The K-sparse Fourier coherence uk (G) and consequently the bounds (111 and (12 1 can be explicitly 
evaluated for some specific orthobases G. For example, letting G = In (the N x N identity matrix), 
we can consider signals that are sparse in the time domain. With this choice of G, one can show 
that UK (/tv) = \J T- another example, we can consider signals that are sparse in the frequency 
domain. To do this, we set G equal to a real-valued version of the Fourier orthobasis. Without loss 
of generality, suppose N is even. The real Fourier orthobasis, denoted Rn, is constructed as follows. 
The first column of equals the first column of F/v. Then R^2 ^} = Real (^■\/2F^2 
R^K^i N-1} — Imaginary (^\/2F^2 -})• The last column of R]\f is equal to the + l)-th column 
of F/v. Similar steps can be taken to construct a real Fourier orthobasis when N is odd. With this choice 
of G, one can show that uk (Rn) < \/^- U^mg these upper bounds on the Fourier coherence, we have 
the following deterministic bounds on p (a) in the time and frequency domains: 

p{a) < K (time domain sparsity) and (13) 
p{a) < N (frequency domain sparsity) . (14) 

We also obtain the following bounds on the expected value of p (a) under the random signal model: 

E [p (a)] < 8 (log 2L + 2) (time domain sparsity) and (15) 

8N 

E [p (a)] < —— (log 2L + 2) (frequency domain sparsity) . (16) 
K 

We offer a brief interpretation and analysis of these bounds in this paragraph and several examples 
that follow. First, because K < N, the deterministic and expectation bounds on p (a) are smaller for 
signals that are sparse in the time domain than for signals that are sparse in the frequency domain. The 
simulations described in Examples [2] and [3] below confirm that, on average, p (a) does indeed tend to be 
smaller under the model of time domain sparsity. Second, these bounds exhibit varying dependencies on 
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the sparsity level K: (13) increases with K and ( [T6| ) decreases with K, while ( 14i and ( [TS] ) are agnostic 
to K. The simulation described in Example |2] below confirms that, on average, p (a) increases with K 
for signals that are sparse in the time domain but decreases with K for signals that are sparse in the 



frequency domain. This actually reveals a looseness in (15i; however, in Section IV-C we conjecture 



a sparsity-dependent expectation bound that closely matches the empirical results for signals that are 
sparse in the time domain. Third, under both models of sparsity, and assuming 8 (log 2L + 2) K for 
signals of practical interest, the expectation bounds on p (a) are qualitatively lower than the deterministic 
bounds. This raises the question of whether the deterministic bounds are sharp. We confirm that this is 
the case in Example |4] below. 

Example 1: (Illustrating the signal-dependency of the left-hand side of CoM inequalities ([T]) and ([8]l) 
In this example, we illustrate that the CoM behavior for randomized Toeplitz matrices is indeed signal- 
dependent. We consider inequality ([7]) while a similar analysis can be made for ([8]l. We consider two 
particular ii'-sparse (K = 64) signals, ai and a2 both in {N = 1024) where the K non-zero entries 
of ai have equal values and occur in the first K entries of the vector (p (ai) = 63.26), while the K non- 
zero entries of a2 appear in a randomly- selected locations with random signs and values (p (02) = 5.47). 
Both ai and 02 are normalized. For a fixed M = 512, we measure each of these signals with 1000 i.i.d. 
Gaussian M x N Toeplitz matrices. Figure [T] depicts the numerically determined rate of occurrence of 
the event ||y||2 — Af||a||2 > eM||a||2 over 1000 trials versus e G (0,1). For comparison, the derived 
analytical bound in ^ (for Toeplitz X) as well as the bound in Q (for unstructured X) is depicted. 
As can be seen the two signals have different concentrations. In particular, ai (Fig. l(a)[ ) has worse 



concentration compared to 02 (Fig. 1(b)) when measured by Toeplitz matrices. This signal-dependency 
of the concentration does not happen when these signals are measured by unstructured Gaussian random 
matrices. Moreover, the derived signal-dependent bound Q successfully upper bounds the numerical event 
rate of occurrence for each signal while the bound (|9]) for unstructured X fails to do so. Also observe 
that the analytical bound e ^^p'^s) in Fig. 



1(b) 



can not bound the numerical event rate of occurrence for 



ai as depicted in Fig. 1(a) 



Example 2: (Varying K and comparing the time and frequency domains) In this experiment, we fix 
M and N. For each value of K and each sparse basis G = In and G = Rn, we construct 1000 random 
sparse vectors q G with random support and having K non-zero entries drawn from a Gaussian 
distribution with mean zero and variance ^. For each vector, we compute a = Gq, and we then let p (a) 
denote the sample mean of p (a) across these 1000 signals. The results, as a function of K, are plotted in 



Fig. 2(a) As anticipated, signals that are sparse in the frequency domain have a larger value of p (a) than 
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Numerical Rate 



0.2 0.4 0.6 0.8 1.0 



(a) 



(b) 



Fig. 1 : Illustrating the signal-dependency of the left-hand side ofCoM inequality (|7|. With fixed N = 1024, K — 64 
and M — 512, we consider two particular K-sparse signals, , and a-^. Both ai and 02 are normalized. We measure 
each of these signals with 1000 i.i.d. Gaussian M x N Toeplitz matrices, (a) The K non-zero entries of Oi have equal 
values and occur in the first K entries of the vector, (b) The K non-zero entries of a2 appear in randomly-selected 
locations with random signs and values. The two signals have different concentrations that can be upper bounded by 
the signal-dependent right-hand side of (|7|. 



signals that are sparse in the time domain. Moreover, p (a) decreases with K in the frequency domain 
but increases with K in the time domain. Overall, the empirical behavior of p (a) is mostly consistent 
with the bounds ([15) and ( [76] ), although our constants may be larger than necessary. 

Example 3: (Varying M and comparing the time and frequency domains) This experiment is identical 
to the one in Example |2j except that we vary M while keeping K and N fixed. The results are plotted 



in Fig. 2(b) Once again, signals that are sparse in the frequency domain have a larger value of p (a) 
than signals that are sparse in the time domain. Moreover, in both cases p (a) appears to increase 



logarithmically with M as predicted by the bounds in (15 1 and (16 1, although our constants may be 
larger than necessary. 

Example 4: (Confirming the tightness of the deterministic bounds) We fix A^, consider a vector a G 
that is K-sparse in the time domain, and suppose the K non-zero entries of a take equal values and 
occur in the first K entries of the vector. For such a vector one can derive a lower bound on p (a) by 
embedding P{a) inside a circulant matrix, applying the Cauchy Interlacing Theorem \23j (we describe 



these steps more fully in Section IV- A 1 1, and then performing further computations that we omit for the 
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Fig. 2: Sample mean of p (a) in the time and frequency domains versus the expectation bounds ([T5| and ([T6|, where 
L — N + M — 1. The sample mean p{a) is calculated by taking the mean over 1000 constructed signals a. A 
logarithmic scale is used for the vertical axis, (a) Example^ Varying K with fixed M — N — 256. (b) Example^ 
Varying M with fixed N = 512 and K = 32. 



sake of space. With these steps, one concludes that for this specific vector a G 

2'^ 




(17) 



When M ^> K^, the right-hand side of (17i approaches K. This confirms that (13 1 is sharp for large 
M. 



In the remainder of the paper, the proofs of Theorem [T] (Section III) and Theorem [2] (Section IV) are 



presented, followed by additional discussions concerning the relevance of the main results. Our results 
have important consequences in the analysis of high-dimensional dynamical systems. We expound on 
this fact by exploring a CBD problem in Section |V] 

III. Proof OF Theorem [U 

The proofs of the upper and lower bounds of Theorem [T] are given separately in Lemmas |2] and [3] 
below. The proofs utilize Markov's inequality along with a suitable bound on the moment generating 
function of \\y\\'2, given in Lemma [T] Observe that for a fixed vector a G and a random Gaussian 
Toeplitz matrix X G M}^^^ , the vector y = Xa G R*^ will be a Gaussian random vector with zero 
mean and M x M covariance matrix P (a) given in ([6]). 
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Lemma 1: If y G M^^ is a zero mean Gaussian random vector with covariance matrix P, then 



E 



v/det (/m - 2tP) 



(18) 



holds for all t G (— oo, i^y^ 
Proof 



E 



1 



M , 1 



(27r)^ dets (P) 
1 



dy 



dett ( (P-I - 2t/M 



(27r)^ det2 (P) 

1 _ 1 

(det (P-i - 2t/Af) detP)^ ^det {Im - 2tP) ' 



dets (P) 



Observe that as a special case of Lemma [T] if y G M is a scalar Gaussian random variable of unit 
variance, then we obtain the well known result of E [e 



^7E==, for t G (— oo, |). Based on Lemmajlj 
we use Chernoff's bounding method |24| for computing the upper and lower tail probability bounds. In 
particular, we are interested in finding bounds for the tail probabilities 



and 



P{\\y\\l>M\\a\\l{l + e)} 



P {\\y\\l < M\\ag{l - e)} 



(19b) 



Observe that in (19a) and (19bi concentration behavior is sought around E |||y||2| = -^||ct|l2- For ^ 



random variable z, and all t > 0, 



P{z>e} = P{e*^ > e*'} < 



E [f 



tz~\ 



(20) 



(see, e.g., f24]). Applying ( [20| ) to (19a), for example, and then applying Lemma [T] yields 



P{||l/||i>M||a||2(l + e)} < 



E 



:,iy y 



,M||a|||{l+e)t 



(det {lM-2tP)y--e 



-\ -M\\a\\l{l+e)t 



(21) 



In (|211), t G (— oo, — ^ , is a free variable which — as in a Chernoff bound — can be varied to make the 

1 • ' \ ' 2(maxi Xi) ' 



right-hand side as small as possible. Though not necessarily optimal, we propose to use t = 2(i-|-g)/(a)||a||2 ; 
where / is a function of a that we specify below. We state the upper tail probability bound in Lemma [2] 
and the lower tail probability bound in Lemma [3] 
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Lemma 2: Let a G be fixed, let P = P (a) be as given in and let y G M.^^ be a zero mean 
Gaussian random vector with covariance matrix P. Then, for any e G (0, 1), 

P{||2/||i>M||a||2(l + e)} <e"^. (22) 



Proof Choosing t as 



2{l + e)p{a)\\a\\l 



and noting that t G (— oo, tt— f — r-), the right-hand side of (|21|) can be written as 



det Im 



(l + e)p(a)||a||2 



e f(° 



This expression can be simplified. Note that 

det \Im " ^ 



(l + e)p(a)||a||2 



M 



n 1 



V (l + e)/>(a)||a||2 



Epilog 1 



(23) 



Using the facts that log(l - C1C2) > C2log(l — ci) for any ci,C2 G [0,1] and Tr (P) = M||a|||, we 



have 



Efiiiog 1 



Tr(-P) 
3 p(<i)||a.|| 



jl°s(lT7)=,;;f£ylog(Ti,) 



1 \ p(") 



1 + e 



Combining (21 1, (23 1, and (24 1 gives us 



P{\\yg>M\\a\\lil + e)}<U-^ 



+ e 



g p(a) 



g-t \ 2p(ci) 



The final bound comes by noting that (1 + e)e < e 

Lemma 3: Using the same assumptions as in Lemma |2| for any e G (0, 1), 

P{\\y\\l<M\\a\\l{l-e)} <e-^K 



(24) 



Proof Applying Markov's inequality to (19bi, we obtain 



E 



-ty y 



P{\\yg < M||a||2(l - e)} = P {-\\yg > -M||a||2(l - e)} < 



M||a||i(l-e)r 



(25) 
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Using Lemma [T] this implies that 

P{\\y\\l< M\\ag{l -e)}< ((det {Im + 2tP))-^ e2|l«ll^(i-)*^ ' 
In this case, we choose 



(26) 



2(l + e)Ma)||a||2 



2 ' 



and note that t > 0. Plugging t into (26 1 and following similar steps as for the upper tail bound, we get 



det(/M + 2tP) = J]fl + - 



1=1 



A,; 



Since log (1 + c) > c — y for c > 0, 

M 



i=l 



A,; 



+ e)fi{a) \\a\\l 



M 



> 



+ e) n{a)\\a\\l 



+ M(a 



(27) 



(^(l + e)^(a)||a||2 2 V (1 + e) /i (a) ||a||2 



Ai 



Ei!iiA. 1 



^l + e)/i(a)||a||^ 2 V (1 + e) ^ (a) ||a||^ 



2 A/ 



i=l 



M 1 



M M / e^ + 2e 



l + €)fi{a) 2\l + €j ii{a) //(a) V2(l + e) 



Combining (27i and (28 1 gives the bound 



det {Im + '^tP) > e^<») V2(i+=)' 



g2(l + £)2 



(28) 



(29) 



By substituting (29 1 into (26 1, we obtain 

P < M\\a\\l (1 - e)} < (^^t^e 

The final bound comes by noting that e < e""^^/^. ■ 

IV. Proof and Discussion of Theorem|2] 

A. Proof of Theorem |2] 

1 ) Circulant Embedding: The covariance matrix P (a) described in ^ is an M x M symmetric 
Toeplitz matrix which can be decomposed as P (a) = A, where A is an {N + M — 1) x M Toeplitz 
matrix (as shown in Fig. [s]) and is the transpose of A. In order to derive an upper bound on the 
maximum eigenvalue of P (a), we embed the matrix A inside its {N + M — 1) x [N + M — 1) circulant 
counterpart Ac where each column of Ac is a cyclic downward shifted version of the previous column. 
Thus, Ac is uniquely determined by its first column, which we denote by 



a = loi 



a^v 

(A/— 1) zeros 



r G M^, 



July 16, 2012 



DRAFT 



14 



A 



ai ... 

as ■•■ (0) 

: '■■ ai : 

Cat 02 ■- 

■•■ : ■•■ ai 

: aN 02 

(0) •-. : 

... OAT 

dLxM 



01 

02 "•■ 

OAT 

"•■ 

(0) 



Oat ... 02 



Ol 

OAT 



(0) 

■•■ Ol 

Oat 



OAT 





■-. 

02 Ol 



Fig. 3: Toeplitz matrix A e M-^^*^ and its circulant counterpart Ac e R^^'^ where L = N + M - 1. 



where L = N -\- M — 1. Observe that the circulant matrix Ac € R^^^ contains the Toeplitz matrix 
A € R^^^^ in its first M columns. Because of this embedding, the Cauchy Interlacing Theorem |23| 
implies that maxj<7v/ Xi{A'^A) < maxj<j^ Xi{A]^Ac). Therefore, we have 

maxj Aj(P (a)) maxiXi^A^A) maxi Xi{A]^ Ac) maxj |Aj(j4^)p 



p{a) 



< 



a 



12 11*^112 II"'II2 II"'II2 

Thus, an upper bound for p (a) can be achieved by bounding the maximum absolute eigenvalue of A^. 
Since Aj is circulant, its eigenvalues are given by the un-normalized length-L Discrete Fourier Transform 
(DFT) of the first row of (the first column of Ac). Specifically, for i = 1, 2, . . . , L, 

L N 

XM^) = j;S,e-¥(-i)(^-i) = j;a,e-¥(-i)(^-i). (31) 



a 



a 



--: pc (a) . (30) 



k=l 



k=l 



Recall that Fl E C^^^ is the Fourier orthobasis with entries FL{i,m) = -^w^^ where w 



, and let FT^ G be the i-th row of F^. Using matrix-vector notation, (31 1 can be written as 



XiiA^) = VlFta = ^Ft^'^a = ^/LFi;^Gq = ^FU^Gsqs, 



(32) 



where Fl~^ G is a row vector containing the first N entries of -F^~*, G R^ is the part of g G R^ 
restricted to the support S (the location of the non-zero entries of q) with cardinality \S\ = K, and 
Gs G R^^-^ contains the columns of G G R^^^ indexed by the support S. 

2) Deterministic Bound: We can bound p (a) over all sparse a using the Cauchy-Schwarz inequality. 



From (32 1, it follows for any i G {1, 2, . . . , L} that 

\Xi{A^)\ = \VLFi;tfGsqs\ < VL\\Fi;tfGsh\\qsh 



-^l|-^l:ArG's'||2||a||2- 



(33) 
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By combining Definition |2j ( [30] ), and (33 1, we arrive at the deterministic bound ( [TT] ). This bound appears 
to be highly pessimistic for most sparse vectors a. In other words, although in Example [4] we illustrate 



that for a specific signal a, the deterministic bound ( [TT] ) is tight when M ^ K, v/e observe that for 
many other classes of sparse signals a, the bound is pessimistic. In particular, if a random model is 



imposed on the non-zero entries of a, an upper bound on the typical value of p (a) derived in ( 15 ) scales 
logarithmically in the ambient dimension L which is qualitatively smaller than K. We show this analysis 
in the proof of Theorem [2] In order to make this proof self-contained, we first list some results that we 
will draw from. 

3) Supporting Results: We utilize the following propositions. 



Lemma 4: \ 19 1 Let z be any random variable. Then 

E[|z|] = / P{|z| > x}dx. 
Jo 



(34) 



Lemma 5: Let zi and Z2 be positive random variables. Then for any U, 



and for any Ui and U2, 



P{2i + ^2>f/}<P<!2i>^|+p|^2>^ 



P{->^} <P{Z1>UI}+P{Z2<U2}. 
Z2 1J2 



(35) 



(36) 



Proof See Appendix A. ■ 
Proposition 1: |3] {Concentration Inequality for Sums of Squared Gaussian Random Variables) Let 



q G M be a random i^-sparse vector whose K non-zero entries (on an arbitrary support S) are i.i.d. 
random variables drawn from a Gaussian distribution with A/'(0, o"^). Then for any e > 0, 

P{||q||i <Ka'^{l-e)] < e"^. 



Proposition 2: (Hoeffding's Inequality for Complex-Valued Gaussian Sums) Let b G be fixed, 
and let e G be a random vector whose N entries are i.i.d. random variables drawn from a Gaussian 
distribution with AA(0, a^). Then, for any u > 0, 



N 

E 

i=l 



> u \ <2e -'"'""lis . 



July 16, 2012 



DRAFT 



16 



Proof See Appendix B. ■ 
In order prove Theorem |2| we also require a tail probability bound for the eigenvalues of A^. 
Proposition 3: Let q G be a random K-sparse vector whose K non-zero entries (on an arbitrary 
support S) are i.i.d. random variables drawn from a Gaussian distribution with Af{0, Let a = Gq 
where G £ R^^^ is an orthobasis, and let Ache an L x L circulant matrix, where the first N entries 
of the first column of Ac are given by a. Then for any u > 0, and for i = 1, 2, . . . , L, 

P{|Ai(^e)| >^^} <2e"^^^. (37) 



Proof Define the row vector b = \/ljF^TpGs S C^. From (32) and the Cauchy-Schwarz inequality, it 
follows that |Aj(Ac)| = |Ai(A^)| = \VLFl;^Gsqs\ = I Y.f=i where et = {qs)i- From Definition [i] 
we have ||6||2 < VLi'k {G). The tail probability bound ( [37] ) follows from applying Proposition [2] ■ 

4) Completing the Proof of Theorem^ From (30 1, we have 

max,|A,(A^)|2l /-^ rmaxi|Ai(^J)|2 



E[p(a)]<E[pe(a)]=E 



a 



a 



> x> dx 



maxj \\i{A 



T\\2 



a 



where the last equality comes from the deterministic upper bound |Ai(^^)| < \/Z||-?^{rjvG5||2||a||2 < 
\fljVK (G) \\a\\2. Using a union bound, for any t > we have 



maxj I Aj {A 



T\|2 



> x> dx 



a 



Lul(G) 







maxj |Ai(^ 



T\\2 



\a\\ 



>jUx 



< I P|max|Ai(^^)p > txjdx 

LuUG) 

P{l|a|l2 <t]dx. 



(38) 
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The first term in tlie riglit liand side of (38 1 can be bounded as follows. For every 5 > 0, by partitioning 



the range of integration |19|, |25|, we obtain 



P|max|Ai(^^)P > txjdx < y P jmax |Ai(A^)|2 > txj 

<(^ + ^ P |max|Ai(^c)P > tej dx 

/■oo 

<(5+ / VP{|Ai(Ac)|2 > txjdx 



j=i 

oo 



6 + 2L e "^"^("^'dx 



s + 



Kt 



where we used Proposition [3] in the last inequality. The second term in p8] ) can be bounded using the 

0r II ii9 1 g(i-t)^ 
We have for < t < 1, P |||a||2 < i| < e « . Putting 

together the bounds for the two terms of inequality p8]), we have 



E [p (a)] < E [pc (a)] <S + 



8^^4 (g) 



e ^^"k"^' + Li/^ (G) e" 



(39) 



Now we pick 5 to minimize the upper bound in (39 1. Using the minimizer (5* = ^^'^ic(^)^°g yields 



E[p(a)] <E[p,(a)] < 



4^4 (g) 
Kt 



Kt K(l-t)^ 

log2L + l + — e 5— 



(40) 



L^tgiK,t) :=f e — 



It is trivial to show that g {K, 0.5) < 1 for all K (for t = 0.5, max^ g {K, 0.5) 



^). Therefore, E [p (a)] < ^^I^M^^ (log2L + 2), which completes the proof. 



B. Discussion 

Remark 1: In Theorem[2j we find an upper bound on E [p (a)] by finding an upper bound on E [pc (a)] 
and using the fact that for all vectors a, we have p{a) < Pc{o). However, we should note that this 
inequality gets tighter as M (the number of columns of A) increases. For small M the interlacing 
technique results in a looser bound. 

Remark 2: By taking G = In and noting that (In) = \f^' ( |40l ) leads to an upper bound on 
E [pc (a)] when the signal a is i^-sparse in the time domain (specifically, E [pc (a)] < 8 (log2L + 2)). 
Although this bound scales logarithmically in the ambient dimension L, it does not show a dependency 
on the sparsity level K of the vector a. Over multiple simulations where we have computed the sample 
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• — • P,: [a) 

--- fiL] 



500 1000 1500 2000 

Signal Length (iV) 

Fig. 4: Empirical results corresponding to Example^ Sample mean of pc (a) in the time domain for full vectors 
a e where M = 256 is fixed. Also plotted is J (L) = log {L), where L = N + M - I. 



mean pc (a), we have observed a linear behavior of the quantity as K increases, and this leads 

us to the conjecture below. Although at this point we are not able to prove the conjecture, the proposed 
bound matches closely with empirical data. 

Conjecture 1: Fix N and M. Let a G be a random K-sparse vector whose K non-zero entries 
(on an arbitrary support S) are i.i.d. random variables drawn from a Gaussian distribution with Af{0, 
Then 

E[Pc {a)] - ^ , 

CiK + C2 

where ci = ^^^^^ for some constant c, and C2 = 1 — ci. 

The conjecture follows from our empirical observation that s ~ ciK + C2 for some constants ci 
and C2, the fact that pc (a) = 1 for K = 1, and the observation that pc (a) ~ clogL when = for 
large N. In the following examples, we illustrate these points and show how the conjectured bound can 
sharply approximate the empirical mean of pc (a). 

Example 5: In this experiment, we fix M = 256 and take G = In- For each value of A^, we construct 
1000 random non-sparse vectors a G whose N entries are drawn from a Gaussian distribution with 
mean zero and variance We let pc (a) denote the sample mean of pc (a) across these 1000 signals. 
The results, as a function of A^, are plotted in Fig. |4] Also plotted is the function / (L) = log (L) where 
L = N + M — 1; this closely approximates the empirical data. 

Example 6: In this experiment, we fix = 1024. For each value of K, we construct 1000 random 
sparse vectors a G with random support and having K non-zero entries drawn from a Gaussian 
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distribution with mean zero and variance ^. We let pc (a) denote the sample mean of pc (a) across these 
1000 signals. The results, as a function of K for two fixed values M = 1 and M = 1024, are plotted in 
Fig.H 

Remark 3: As a final note in this section, the result of Theorem [2] can be easily extended to the case 
when G G C^^^ is a complex orthobasis and q and a are complex vectors. The bounds can be derived 
in a similar way and we do not state them for the sake of saving space. 

C. A Quadratic RIP Bound and Non-uniform Recovery 

An approach identical to the one taken by Baraniuk et al. Q can be used to establish the RIP for 
Toeplitz matrices X based on the CoM inequalities given in Theorem [T] As mentioned in Section |llj 
the bounds of the CoM inequalities for Toeplitz matrices are looser by a factor of 2p (a) or 2p (a) as 
compared to the ones for unstructured X. Since p (a) is bounded by K for all if-sparse signals in the time 
domain (the deterministic bound), with straightforward calculations a quadratic estimate of the number 
of measurements in terms of sparsity (M ~ K^) can be achieved for Toeplitz matrices. As mentioned 
earlier, on the other hand, there exists an extremely non-uniform distribution of p (a) over the set of all 
K-sparse signals a, for as Theorem |2] states, if a random model is imposed on a, an upper bound on 
the typical value of p (a) scales logarithmically in the ambient dimension L. This suggests that for most 
K-sparse signals a the value of p (a) is much smaller than K (observe that 8 (log 2L + 2) <^ K for many 
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N+M 



r™, ^N+M-l 



y = Xa + z 



Fig. 6: FIR filter of order N with impulse response {ak}^^i- 



signals of practical interest). Only for a very small set of signals does the value of p (a) approach the 
deterministic bound of K. One can show, for example, that for any K-sparse signal whose K non-zero 
entries are all the same, we have p (a) < pc (a) = K (Example |4]|. This non-uniformity of p (a) over the 
set of sparse signals may be useful for proving a non-uniform recovery bound or for strengthening the 
RIP result; our work on these fronts remains in progress. Using different techniques than pursued in the 
present paper (non-commutative Khintchine type inequalities), a non-uniform recovery result with a linear 
estimate of M in K up to log-factors has been proven by Rauhut (9). For a detailed description of non- 
uniform recovery and its comparison to uniform recovery, one could refer to a paper by Rauhut [Sections 



3.1 and 4.2, 1 19 1]. The behavior of p (a) also has important implications in the binary detection problem 



which we discuss in the next section. 

V. Compressive Binary Detection 

A. Problem Setup 

In this section, we address the problem of detecting a change in the dynamics of a linear system. We 
aim to perform the detection from the smallest number of observations, and for this reason, we call this 
problem Compressive Binary Detection (CBD). 

We consider an FIR filter with a known impulse response a = {ak}^^i- The response of this filter to 
a test signal x = {xk}^^^^~^ is described in We suppose the output of this filter is corrupted by 
random additive measurement noise z. Fig. [6] shows the schematic of this measurement process. 

From a collection of M measurements y with M < N, our specific goal is to detect whether the 
dynamics of the system have changed to a different impulse response b = {bk}^^i, which we also 
assume to be known. Since the the nominal impulse response a is known, the expected response Xa 
can be subtracted off from y, and thus without loss of generality, our detection problem can be stated as 



follows |26|: Distinguish between two events which we define as £q = {y = z} and £i = {y = Xc + z}, 
where c = b — a and z is a vector of i.i.d. Gaussian noise with variance a^. 
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For any detection algorithm, one can define the false-alarm probability PpA — P {("^i chosen when £q)} 
and the detection probability Pd = P{(<?i chosen when £i)}. A Receiver Operating Curve (ROC) is a 

plot of Pd as a function of Pfa- A Neyman-Pearson (NP) detector maximizes Pd for a given limit on 

£i 

the failure probability, PpA < a- The NP test for our problem can be written as y Xc ^ 7, where the 

£0 

threshold 7 is chosen to meet the constraint PpA < ct. Consequently, we consider the detection statistic 
d := Xc. By evaluating d and comparing to the threshold 7, we are now able to decide between the 
two events £q and £1. To fix the failure limit, we set PpA = ct which leads to 

P^(a) = g('g-i(a)-^^V (41) 



where Q{q) = evident from (41 1, for a given a, Pd (a) directly depends on 

||Xc||2. On the other hand, because X is a compressive random Toeplitz matrix. Theorem [T] suggests 
that 1 1-^^0112 is concentrated around its expected value with high probability and with a tail probability 
bound that decays exponentially in M divided by p (c). Consequently, one could conclude that for fixed 
M, the behavior of p (c) affects the behavior of Pd (a) over a. The following example illustrates this 
dependency. 

Example 7: (Detector Performance) Assume with a failure probability of PpA = a = 0.05, a detection 



probability of Pd (a) = 0.95 is desired. Assume a = 0.3. From (41 1 and noting that Q (—1.6449) = 0.95 
and Q-^ (0.05) = 1.6449, one concludes that in order to achieve the desired detection, ||Xc||2 should 
exceed 0.3 x 2 x 1.6449 = 0.9869 (i.e., \\Xc\\l > 0.9741). On the other hand, for a Toeplitz X with 
i.i.d. entries drawn from A/" (O, jj), E [HXcUl] = ||c||2. Assume without loss of generahty, ||c||2 = 1. 
Thus, from Theorem [T] and the bound in (jsj, we have for e G (0, 1) 

P {||Xc||2 - 1 < -e} < e"^ < e"*^. (42) 



Therefore, for a choice of e = 1 — 0.9741 = 0.0259 and from (|42l), one could conclude that 

P{||Xc||| < 0.9741} < e" 



Consequently, for ( G (0, 1), if M > g 7x^10-4 log then with probability at least 1— 11-^^^112 exceeds 
0.9741, achieving the desired detection performance. Apparently, M depends on p (c) and qualitatively, 
one could conclude that for a fixed M, a signal c with small p (c) leads to better detection (i.e., maximized 
Pd (a) over a). Similarly, a signal c with large p (c) is more difficult to reliably detect. 

In the next section, we examine signals of different p (c) values and show how their ROCs change. It 
is interesting to note that this dependence would not occur if the matrix X were unstructured (which, of 
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Failure Probability (a) Failure Probability {a) 

(a) (b) 

Fig. 7: ROCs for 1000 random matrices X for a fixed signal c witli p (c) = 45.6. (a) Unstructured X. (b) Toeplitz 
X. The solid black curve is the average of 1000 curves. 



course, would not apply to the convolution-based measurement scenario considered here but is a useful 
comparison) as the CoM behavior of unstructured Gaussian matrices is agnostic to the signal c. 

B. Empirical Results and ROCs 

In several simulations, we examine the impact of p (c) on the detector performance. To begin, we 
fix a signal c G M^^^ with 50 non-zero entries all taking the same value; this signal has ||c||2 = 1 
and p (c) = 45.6 with our choice of M = 128. We generate 1000 random unstructured and Toeplitz 
matrices X with i.i.d. entries drawn from N (O, jj). For each matrix X, we compute a curve of Pd over 
PpA using ( [4T] ); we set a = 0.3. Figures |7ja) and|7jb) show the ROCs resulting from the unstructured 
and Toeplitz matrices, respectively. As can be seen, the ROCs associated with Toeplitz matrices are 
more scattered than the ROCs associated with unstructured matrices. This is in fact due to the weaker 
concentration of ||Xc||2 around its expected value for Toeplitz X (recall (j7]l and ([S])) as compared to 
unstructured X (recall (|9])). 

To compare the ROCs among signals having different p (c) values, we design a simulation with 6 



different signals. Each signal again has ||c||2 = 1, and we take a = 0.3 as above. Figures 8(a) and 8(b) 
plot the average ROC for each signal over 1000 random unstructured and Toeplitz matrices, respectively. 
Two things are evident from these plots. First, the plots associated with Toeplitz matrices show a signal 
dependency while the ones associated with unstructured matrices are signal-agnostic. Second, with regards 
to the plots associated with Toeplitz X, we see a decrease in the curves (i.e., inferior detector performance) 
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0.2 0.4 O.C 0.8 1.0 0.2 0.4 O.G 0.8 1.0 

Failure Probability (a) Failure Probability (a) 

(a) (b) 



Fig. 8: Average ROCs over 1000 random matrices X for 6 different signals c. (a) Unstructured X. All curves are 
overlapping, (b) Toeplitz X. The curves descend in the same order they appear in legend box. 



for signals with larger values of p{c). 

In summary, our theory suggests and our simulations confirm that the value of p{c) has a direct 
influence on the detector performance. From a systems perspective, as an example, this means that 
detecting changes in systems having a sparse impulse response in the time domain (e.g., communication 
channels with multipath propagation) will be easier than doing so for systems having a sparse impulse 
response in the frequency domain (e.g., certain resonant systems). It is worth mentioning that while 
detection analysis of systems with sparse impulse response is interesting, our analysis can be applied to 
situations where neither the impulse responses a and b nor the change c are sparse. 
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Appendix 

A. Proof of Lemma |5] 



Proof We start by proving a more general version of (35 1. Let zi, Z2, ■ ■ ■ , Zn^^ random variables. Con- 
sider the event £a = {zi < ciU and Z2 < C2U and • • • Zn < CnU} where ci, C2, . . . , c„ are fixed num- 
bers that sum to 1. It is trivial to see that if £a happens, then the event £b = {zi + Z2 + ■ ■ ■ + Zn < U} 
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must also occur. Consequently, P {{Sb)'^} < P{(<?a)^}> where 

{£aT = {zi > ciU or Z2 > C2U or • • • Zn > CnU} . 

Using the union bound, we have P {{SaY} < P {^^i > ciU} + P {^2 > C2U} H + P {^n > c„C/} 

which completes the proof. The inequality ( [35] ) is a special case of this result with ci = C2 = 0.5. 

We follow a similar approach for proving ( [36] ) where zi and Z2 are positive random variables. 
Consider the event £a — {^1 < Ui and Z2 > U2}- If £a occurs, then "^B — < ^| must also occur. 
Consequently, P {{SbY} < P{{£aY}, where {£aY = {zi > Ui or Z2 < C/2}- Using the union bound, 
we have P {{SaT} < P {^i > ^^i} + P {^2 < U2}. ■ 



B. Proof of Proposition [2[ 

Before proving Proposition [2] we state the following lemma. 

Lemma 6: {Hoeffding's inequality for real-valued Gaussian sums) Let b € be fixed, and let e G 
be a random vector whose entries are i.i.d. random variables drawn from a Gaussian distribution with 
A/'(0,cr2). rpj^gj^^ fQj. ^jjy > 0, 



N 



j=l 



> n S < e 



Proof First note that the random variable XliLi ^i^i is also Gaussian with distribution M (O, o"^||?>||2)- 
Applying a Gaussian tail bound to this distribution yields the inequality p7| . ■ 
Using the result of Lemma [6] we can complete the proof of Proposition [2] 

Proof Let 6j = r j + gfjj Vi where rj is the real part of hi and qi is the imaginary part. Then we have 



N 



2^ eiTi I + I 2^ mi I J 

^i=l / Vi=l / 



> \\b\\2U 
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where the first inequaUty uses Lemma jsj and the last inequality uses Lemma |6|and the facts that > 1 
and 1% > 1. ■ 

References 

[1] M. Ledoux, The concentration of measure phenomenon. Amer Mathematical Society, 2001. 

[2] S. Dasgupta and A. Gupta, "An elementary proof of a theorem of Johnson and Lindenstrauss," Random Structures & 

Algorithms, vol. 22, no. 1, pp. 60-65, 2003. 
[3] D. Achlioptas, "Database-friendly random projections: Johnson-Lindenstrauss with binary coins," Journal of Computer and 

System Sciences, vol. 66, no. 4, pp. 671-687, 2003. 
[4] W. Johnson and J. Lindenstrauss, "Extensions of lipschitz mappings into a Hilbert space," Contemporary mathematics, 

vol. 26, pp. 189-206, 1984. 

[5] E. Candes and M. Wakin, "An introduction to compressive sampling," IEEE Signal Processing Magazine, vol. 25, no. 2, 
pp. 21-30, 2008. 

[6] E. Candes and T. Tao, "Decoding via linear programming," IEEE Trans. Inform. Theory, vol. 51, no. 12, pp. 4203-4215, 
2005. 

[7] R. Baraniuk, M. Davenport, R. DeVore, and M. Wakin, "A simple proof of the restricted isometry property for random 

matrices," Constructive Approximation, vol. 28, no. 3, pp. 253-263, 2008. 
[8] S. Mendelson, A. Pajor, and N. Tomczak-Jaegermann, "Uniform uncertainty principle for Bernoulli and subgaussian 

ensembles," Constructive Approximation, vol. 28, no. 3, pp. 277-289, 2008. 
[9] H. Rauhut, "Circulant and Toeplitz matrices in compressed sensing," Proc. SPARS, vol. 9, 2009. 
[10] J. Haupt, W. Bajwa, G. Raz, and R. Nowak, "Toeplitz compressed sensing matrices with applications to sparse channel 

estimation," IEEE Trans. Inform. Theory, vol. 56, no. 11, pp. 5862-5875, 2010. 
[11] J. Romberg, "Compressive sensing by random convolution," SIAM Journal on Imaging Sciences, vol. 2, no. 4, pp. 1098- 
1128, 2009. 

[12] B. M. Sanandaji, T. L. Vincent, and M. B. Wakin, "Concentration of measure inequalities for compressive Toeplitz matrices 
with applications to detection and system identification," Proc. of 49th IEEE Conf. on Decision and Control, pp. 2922-2929, 
2010. 

[13] J. Y. Park, H. L. Yap, C. J. Rozell, and M. B. Wakin, "Concentration of measure for block diagonal matrices with 
applications to compressive signal processing," IEEE Transactions on Signal Processing, vol. 59, no. 12, pp. 5859-5875, 
2011. 

[14] M. B. Wakin, B. M. Sanandaji, and T. L. Vincent, "On the observability of linear systems from random, compressive 

measurements," Proc. of 49th IEEE Conf. on Decision and Control, pp. 4447-4454, 2010. 
[15] W. Bajwa, J. Haupt, G. Raz, and R. Nowak, "Compressed channel sensing," Proc. of 42nd Annual Conf. on Inform. 

Sciences and Systems (CISS08), pp. 5-10, 2008. 
[16] B. M. Sanandaji, T. L. Vincent, M. B. Wakin, R. Toth, and K. Poolla, "Compressive system identification of LTI and 

LTV ARX models," Proc. of the 50th IEEE Conference on Decision and Control and European Control Conference, pp. 

791-798, 2011. 

[17] J. Tropp, M. Wakin, M. Duarte, D. Baron, and R. Baraniuk, "Random filters for compressive sampling and reconstruction," 
Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), vol. 3, pp. 872-875, 2006. 



July 16, 2012 



DRAFT 



26 



[18] W. Bajwa, J. Haupt, G. Raz, S. Wright, and R. Nowak, "Toeplitz-structured compressed sensing matrices," lEEE/SP lAth 

Workshop on Statistical Signal Processing, pp. 294—298, 2007. 
[19] H. Rauhut, "Compressive sensing and structured random matrices," Theoretical Foundations and Numerical Methods for 

Sparse Recovery, vol. 9, pp. 1-92, 2010. 
[20] H. Rauhut, J. Romberg, and J. A. Tropp, "Restricted isometrics for partial random circulant matrices," Appl. Comput. 

Harmon. Anal, 2011. 

[21] F. Krahmer, S. Mendelson, and H. Rauhut, "Suprema of chaos processes and the restricted isometry property," 2012. 
[22] H. L. Yap and C. J. Rozell, "On the relation between block diagonal matrices and compressive ToepUtz matrices," October 
2011, technical Report. 

[23] S. Hwang, "Cauchy's interlace theorem for eigenvalues of Hermitian matrices," The American Mathematical Monthly, vol. 

Ill, no. 2, pp. 157-159, 2004. 
[24] G. Lugosi, "Concentration-of-measure inequalities," Lecture Notes, 2004. 

[25] M. Ledoux and M. Talagrand, Probability in Banach Spaces: isoperimetry and processes. Springer, 1991. 

[26] M. A. Davenport, R T. Boufounos, M. B. Wakin, and R. G. Baraniuk, "Signal processing with compressive measurements," 

IEEE Journal of Selected Topics in Signal Processing, vol. 4, no. 2, pp. 445-460, 2010. 
[27] J. Tropp and A. Gilbert, "Signal recovery from random measurements via orthogonal matching pursuit," IEEE Trans. 

Inform. Theory, vol. 53, no. 12, pp. 4655^666, 2007. 



July 16, 2012 



DRAFT 



